CKS: check and renew the control plane certificates - #14651

Description

kubeadm issues the control plane certificates of a CKS cluster for one year. CKS never renewed them and never checked their expiry, so a cluster that isn't upgraded within a year stops working when they expire. And because CKS only read the kubeconfig that getKubernetesClusterConfig returns when the cluster started, even an upgrade, which renews the certificates, left it handing out the old admin certificate until that expired.

This PR checks the expiry, alerts ahead of it, adds an API to renew the certificates one control node at a time, renews them automatically before they expire, and refreshes the stored kubeconfig after every upgrade and renewal.

Fixes #14650

What changed

1. Stored kubeconfig refresh. kubeConfigData is read again from the control node:

   • after upgradeKubernetesCluster (a failure there is logged and doesn't fail the upgrade);
   • after every certificate rotation;
   • whenever the expiry check finds the stored copy's admin certificate expiring before the one on the cluster, e.g. after a renewal by hand.

   The read-and-rewrite code that StartWorker had is now KubernetesClusterActionWorker.refreshKubeConfigData(), shared by all three. KubernetesClusterUtil.getKubernetesClusterConfig now waits 5 seconds between attempts instead of retrying a refused SSH connection in a tight loop until its timeout.

2. Expiry check. A new scanner, on its own thread next to KubernetesClusterStatusScanner, runs kubeadm certs check-expiration over SSH on every control node of each Running or Alert cluster, at most every cloud.kubernetes.cluster.certificate.check.interval hours. It also reads the client certificate of /root/.kube/config, which CKS uses for its own kubectl calls. The scanner wakes every 15 minutes, but the per-cluster interval is tracked in a cluster detail, so a cluster is checked once per interval whichever management server runs it, and an unreachable cluster isn't retried until the next interval.

   • When the first leaf certificate reaches one of the cloud.kubernetes.cluster.certificate.expiry.alert.days thresholds, CKS raises an alert (new type ALERT.KUBERNETES.CLUSTER.CERTIFICATE) and records a KUBERNETES.CLUSTER.CERTIFICATES.EXPIRY event. Each threshold fires once per expiry date, and expiry itself alerts once more. A renewal moves the date, so the thresholds start over.
   • The first certificate authority to expire is tracked the same way, with its own alert. CAs are reported, never rotated.
   • When none of a cluster's control nodes can be checked, or the cluster has no address to reach it on, an alert says so, at most once a day, so an unreachable cluster can't expire unnoticed. Each cluster's state is read again just before it is checked, so a cluster stopped during a pass is skipped.

3. rotateKubernetesClusterCertificates API (async, same roles as upgradeKubernetesCluster). For each control node, one at a time, it copies rotate-kubernetes-certificates.sh to the node and runs it. The script:

   • puts back any static pod manifest left aside by an interrupted earlier run;
   • fails before touching anything if crictl can't list the containers, since it needs it to confirm the static pods restarted;
   • waits up to 3 minutes for the local API server to be ready, because kubeadm certs renew silently falls back to its default configuration when it can't read the cluster's. If it still isn't ready, the script warns and renews anyway: once the certificates have expired the API server never becomes ready, and renewing is the way out;
   • runs kubeadm certs renew all, and fails if apiserver.crt or admin.conf did not change or kubeadm says a certificate "can't be renewed" (external CA mode, where kubeadm exits 0 without renewing anything);
   • copies admin.conf to /root/.kube/config. If that file was something else before the renewal, it is kept as config.cks-backup, and an existing backup is never overwritten;
   • moves the kube-apiserver, kube-controller-manager, kube-scheduler and etcd manifests out, waits until the kubelet has stopped those containers, then moves the manifests back. An exit trap moves them back on any failure too;
   • waits for /readyz on the API server, /healthz on the controller manager and scheduler, and for the API server to serve the renewed certificate.

   Once the script is done on a node, the worker waits for the cluster API endpoint to answer and the node to be Ready before it moves to the next node. The shared readiness check now also accepts Ready,SchedulingDisabled, so a cordoned control node doesn't fail the rotation. It stops at the first node that fails, moves the cluster to Alert and raises an alert. After the last node it refreshes the stored kubeconfig.

   A failed rotation leaves the cluster in Alert, and the API accepts Alert as well as Running, so it can be retried right away. A rotation started from Alert goes back to Alert when it succeeds. Whatever else put the cluster there hasn't been checked, so the status scanner moves it to Running once its nodes are ready.

   The API also rescues a cluster whose certificates have already expired. Its API server no longer answers, so it is in Alert. With stacked etcd and several control nodes, one renewed node can't get etcd quorum until the others are renewed too, so waiting for each node in turn would never get past the first. When kubeadm certs check-expiration on any control node shows an expired certificate that the control plane runs on, the worker switches to rescue mode. An expired admin.conf or super-admin.conf breaks no component and doesn't count. Checking every node matters: after an interrupted rescue the first node is renewed while the others still block quorum. The decision goes by expiry, not by whether the API answers. An expired API server that is still running keeps answering /version, while a healthy cluster that is briefly unreachable must keep the node-by-node path, since restarting every control node at once would cost it etcd quorum. In rescue mode:

   • it renews every control node first, and on each node the script only waits for etcd (when the node runs one) to run again and for new sandboxes of the other control plane pods. The API server keeps restarting until etcd has quorum, so it can't wait for that;
   • then it waits for the cluster API and for every control node to be Ready.

   Rerunning is safe. kubeadm certs renew issues fresh certificates every time, and the interrupted-run recovery above makes the script pick up where a killed run left off.

4. Automatic renewal. When the first renewable leaf certificate is within cloud.kubernetes.cluster.certificate.auto.renew.days and cloud.kubernetes.cluster.certificate.auto.renew.enabled is true for the cluster's account, the scanner hands the same rotation to a small thread pool, so a renewal that takes minutes doesn't hold up the checks of other clusters. When it is done, an event tells users to download the kubeconfig again. Values above 180 days are treated as 180, so a mistyped setting can't renew every cluster on every check.

   • Certificates kubeadm reports as externally managed are left out of this decision, since renewing can't fix them and every check would renew again.
   • After a failed rotation, automatic renewal waits 24 hours before trying again; the failure has already raised an alert.
   • When a control node couldn't be checked, automatic renewal waits too: it would renew the reachable nodes and then fail on that one. The expiry alert then says renewal is on hold and names the nodes.

5. Guards.

   • Rotation and the check need Kubernetes 1.20 or later, where kubeadm certs replaced kubeadm alpha certs. The API refuses older clusters and the scanner skips them.
   • The API refuses clusters that aren't Running or Alert, and externally managed clusters. Clusters in Alert are checked and alerted on, since one can sit there for long (a stopped worker is enough), but automatic renewal only runs on Running clusters.
   • A queued automatic renewal checks again when it gets a thread, and skips the cluster if it is no longer Running or was rotated meanwhile, successfully or not.
   • Clusters created before 4.16 have no control node flag in kubernetes_cluster_vm_map. For those, the control nodes are taken by position, after any etcd nodes, as fetchControlVmIfMissing already does.
   • startKubernetesCluster and stopKubernetesCluster refuse a cluster in RotatingCertificates, whose control plane manifests may be moved aside at that moment. Upgrade already requires Running, and scale Created, Running or Stopped.
   • A new transient state, RotatingCertificates, is entered from Running or Alert through the new CertificateRotationRequested event. It goes back to Running on success and to Alert on failure. As with upgrade and scale, the state transition is what keeps other operations out: none of them can start from RotatingCertificates, and a rotation can't start while another operation runs.
   • The status scanner's Alert reconciliation could move a cluster to Running, or to Stopped, while a rotation from Alert had just started. reconcileAlertCluster and the all-VMs-stopped branch now stop when their own first transition fails, instead of applying OperationSucceeded regardless.
   • On management server shutdown, automatic renewals still waiting for a thread are dropped rather than started; the next check after the restart schedules them again.
   • If the management server running a rotation goes away, the cluster would stay in RotatingCertificates. The rotation records which management server runs it. When that server comes back, its scanner moves the rotation to Alert straight away; any other server does so once it is well past cloud.kubernetes.cluster.certificate.rotation.timeout (twice the timeout plus 10 minutes). Either way an alert is raised.

6. upgrade-kubernetes.sh now also copies the renewed admin.conf to /root/.kube/config on control nodes, so CKS's own kubectl keeps working after the old certificate expires. As in the rotation script, a /root/.kube/config that differed from admin.conf before the upgrade is kept as a backup first.

No schema changes. The state column is char(32), and the bookkeeping (last check, alerted thresholds, rotation start and last failure) is kept in kubernetes_cluster_details.

Settings

All of them are dynamic.

Automatic renewal is on by default. A plain kubeadm cluster with no upgrades and no renewal timer is the one setup where the certificates simply run out. RKE2 and k3s renew on restart, and managed services rotate them unseen. Renewing 30 days ahead leaves a month for people and CI that downloaded the old kubeconfig to fetch it again, and the event says so. Anyone who wants to schedule the restarts themselves can turn it off globally or per account and use the API.

Restart impact

Static pods don't reload certificates, so each control node's API server, controller manager, scheduler and stacked etcd restart. One node takes about 40 seconds.

• Several control nodes: the restart rolls one node at a time and etcd keeps quorum. The API as a whole stays up behind the load balancer, but requests the load balancer sends to the restarting node fail until its API server is back; clients retry them.
  • How many fail depends on the load balancer. With kind's load balancer, 6 of 187 polls failed, two per node, because its health check takes the node out quickly.
  • With the CloudStack network load balancer in front of a 3-control-node CKS cluster, about a third of requests failed for roughly 20 s per node, because it kept sending to the restarting node: 42 of 491 polls across two rotations.
  • The cloud controller manager pod may restart once per rotation when it loses its leader lease. It recovers on its own.
• One control node: the API is unavailable for roughly 30–40 seconds.
• Workloads keep running in both cases. Only the control plane restarts; kubelets and pods don't.
• Kubeconfigs downloaded before the renewal keep working until their old certificate expires. Renewal doesn't revoke the old certificate.

External etcd

kubeadm certs renew doesn't touch separate etcd nodes, but the etcd nodes CKS creates (etcd-node.yml) serve etcd over plain HTTP: --listen-client-urls http://..., peer URLs http://..., no --cert-file. They hold no certificates, so there is nothing to renew or check on them, and the rotation only touches control nodes. kubeadm sees those clusters as having external etcd, so check-expiration and renew leave out the etcd certificates there and handle the rest normally.

If etcd nodes ever get TLS, they'll need their own renewal step. Unencrypted etcd traffic is a separate concern and out of scope here.

Not in this PR

• CA rotation; CAs are only reported.
• A UI button for the new API.
• /etc/kubernetes/user.conf. CKS returns it instead of admin.conf when an operator has put one on the control node. It is the operator's file, possibly signed by something CKS doesn't manage, so CKS neither renews nor rewrites it. The stale-kubeconfig check compares the stored copy with it rather than with admin.conf.
• The cluster state update is not a compare-and-set (KubernetesClusterDaoImpl.updateState writes unconditionally). That's how every CKS operation works today, and this PR uses the same guard. Likewise, the existing workers carry on when their own first transition fails. An upgrade validated as Running just before an automatic renewal takes the cluster could therefore still run alongside it; the window is the time between the API's state check and the worker's transition. Making updateState conditional, and having every worker stop when its first transition fails, would close this race for all operations. That's a separate change.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

Unit tests

All 189 tests of the CKS plugin pass, including these new ones.

• KubernetesClusterCertificateUtilTest:
  • parsing real kubeadm certs check-expiration output (v1.31): both tables, missing, expired and externally managed certificates, and error output;
  • reading a kubeconfig's client certificate expiry;
  • the threshold parsing and alert decision (one alert per threshold per expiry date; a cluster first seen inside both windows alerts once; expiry alerts once more);
  • the auto-renew decision.
• KubernetesClusterCertificateWorkerTest:
  • the rolling sequence: node by node, renew then wait for ready, then the kubeconfig refresh and Running;
  • stopping at the first node that fails to renew or doesn't become ready, with the remaining nodes untouched, Alert and an alert;
  • rescue mode when a certificate on any control node has expired (every node renewed first, then each waited for), and the node-by-node path otherwise;
  • a rotation from Alert going back to Alert;
  • a second rotation of the same cluster refused without disturbing the first, and a cluster another server moved out of this rotation meanwhile left in its state;
  • /root/.kube/config treated as externally managed when admin.conf is (external CA mode), so it doesn't trigger renewal;
  • failure events recorded only for automatic renewals, since the API's async job records its own;
  • a failed kubeconfig refresh;
  • refusal when the state transition fails and before 1.20;
  • the per-node SSH endpoint (direct network, the cluster IP's port forwarding rule rather than a user's rule on another IP, and by position);
  • the check across control nodes with one unreachable;
  • refreshing a stale stored kubeconfig, and comparing it with user.conf when there is one.
• KubernetesClusterCertificateManagementTest:
  • the scanner: at most one check per interval, the 60 and 30 day alerts firing once each as expiry approaches, thresholds starting over after a renewal, CA alerts without rotation, Alert clusters checked but not renewed, no automatic renewal while a control node is unreachable (and the alert saying so), the daily alert when no node can be checked or the cluster can't be reached at all, the alert advice using the effective renewal days, auto-renew on and off and handed to the rotation thread pool (queued once, requeued after a rejection, skipped when renewed meanwhile), the 24 hour back-off after a failure, externally managed certificates not triggering renewal, and the version gate;
  • moving an interrupted rotation to Alert: at once when it belongs to this management server, otherwise only once it is stale, and never a rotation that just finished;
  • the API validation (Running and Alert accepted) and the new state machine transitions.
• KubernetesClusterUpgradeWorkerTest: the stored kubeconfig is refreshed after a successful upgrade, before the cluster goes back to Running; a failed refresh doesn't fail the upgrade; a failed upgrade doesn't refresh.

The renewal script on real kubeadm control planes

I ran rotate-kubernetes-certificates.sh on throwaway kind clusters (Kubernetes v1.31, kubeadm with stacked etcd):

• Single control node:
  • certificates renewed, all four static pods restarted with new container IDs, the API server serving the renewed certificate, /root/.kube/config identical to the new admin.conf, the node Ready. About 40 seconds.
  • ca.key moved away (external CA mode): kubeadm exits 0 and renews nothing it needs the CA key for. The script fails before restarting anything, and the API server container is unchanged.
  • Killed half-way, with the kube-apiserver and etcd manifests left aside and the API server down: the rerun puts them back, waits for the API server, renews with the cluster's own kubeadm configuration rather than the fallback defaults, and completes.
  • ca.crt hidden so kubeadm certs renew all itself fails: the script prints kubeadm's error, exits 1, and leaves all four manifests in place.
  • API server unreachable (port 6443 rejected inside the node, standing in for expired certificates): the script waits 3 minutes, warns, renews, restarts the static pods and completes once the API server is reachable again.
• Rescue of a three-node cluster with no etcd quorum. I rejected the etcd peer port on all three nodes, standing in for expired peer certificates. That leaves etcd without quorum and the API servers restarting. This confirmed the review's point: a node-by-node wait can't get past the first node.
  • In rescue mode, each node took about 40 seconds, and the script exited 0.
  • Once the peer port was open again, /readyz returned ok and all three nodes were Ready. Each API server served its renewed certificate, and etcd reported all three members healthy.
  • Without the cluster's API, kubeadm renews with its default configuration. It takes the certificate attributes (CN, SANs) from the existing certificates, so they come out the same.
• A custom /root/.kube/config survived two renewals in a row as config.cks-backup, and the readiness check matched a cordoned control node.
• Three control nodes behind kind's load balancer: renewed node by node, waiting for each node to be Ready with the same kubectl get nodes check the worker uses, while polling /readyz through the load balancer every 0.5 s.
  • Every node served its renewed certificate afterwards, and etcd reported all three members healthy.
  • 6 of 187 load balancer polls failed, two per node while its API server restarted.

I also fed the check-expiration output and the /root/.kube/config certificate line from those nodes through the Java parser, and the expiry dates matched.

I haven't run a full CKS cluster through the new API and the scanner end to end. The Java orchestration around the script is covered by the unit tests above.